In the summer, relax on the beaches at Fort Worden. You can camp there too either on the beach or furter inland.
See a movie at the Rose Theatre, which gets mostly indie films that only open in certain theatres.See a movie at the Uptown Theatre.
In mid to late September Port Townsend has it's film festival where there's movies indoors and outdoors during the night. This is one of the biggest independent film festivals in the world.
Stroll around on the downtown strip, visiting the interesting shops.
Port Townsend also holds a Wooden Boat Festival also in September.